Skip to content

[fix](routine load) Restore job properties after FE restart - #69929

Closed
Aranya7 wants to merge 1 commit into
apache:masterfrom
Aranya7:fix/routine-load-config-recovery
Closed

Aranya7 wants to merge 1 commit into
apache:masterfrom
Aranya7:fix/routine-load-config-recovery

Conversation

@Aranya7

@Aranya7 Aranya7 commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #69928

Related PR: N/A

Problem Summary:

Routine Load jobs can silently revert to default execution properties after a Frontend (FE) restart or metadata recovery, even when nondefault values were explicitly configured during job creation.

The affected properties include:

  • max_filter_ratio
  • exec_mem_limit
  • send_batch_parallelism
  • load_to_single_tablet

These properties are correctly initialized when a job is created. However, their corresponding runtime fields are not independently serialized, and the existing metadata recovery path does not restore them from the persisted configuration.

As a result, a successfully recovered Routine Load job may execute with settings that differ from its user-defined configuration, without an explicit ALTER ROUTINE LOAD operation.

Impact

This issue can affect ingestion correctness and operational reliability:

  • Data-quality enforcement: max_filter_ratio can revert from a strict value such as 0.0 to the default 1.0, allowing a substantially higher proportion of filtered rows to be tolerated than intended.
  • Resource management: exec_mem_limit can revert to its default, potentially changing task memory consumption or causing unexpected memory-limit failures.
  • Ingestion performance: send_batch_parallelism can revert to its default, changing batch submission parallelism and throughput.
  • Tablet-loading behavior: load_to_single_tablet can revert to false, changing the configured loading strategy.

The primary concern is that an FE restart can silently alter ingestion semantics. This PR addresses that risk; it does not claim that a specific production data-loss incident has occurred.

Root Cause

RoutineLoadJob maintains the affected configuration values in runtime fields that are not independently persisted in FE metadata.

Although some of these values are available in the serialized jobProperties map, gsonPostProcess() reconstructs the Routine Load description without restoring the corresponding runtime fields.

Additionally, exec_mem_limit was not previously recorded in jobProperties during job creation.

Consequently, metadata deserialization can reconstruct a valid job object while leaving its execution properties initialized to defaults.

Changes

  1. Persist the effective execution memory limit

    • Store the resolved exec_mem_limit in jobProperties during Routine Load job creation.
  2. Restore runtime configuration during metadata recovery

    • Restore max_filter_ratio, exec_mem_limit, send_batch_parallelism, and load_to_single_tablet during gsonPostProcess().
    • Apply the following recovery precedence:
      1. Persisted jobProperties, including values modified through ALTER.
      2. Original CREATE statement, for properties missing from older metadata.
      3. Default configuration, when neither source provides a value.
  3. Preserve ALTER semantics

    • Persisted configuration takes precedence over the original CREATE statement, which may no longer reflect the job's latest settings.
    • For example, if max_filter_ratio was initially 0.5 and subsequently altered to 0.0, the recovered value remains 0.0.
  4. Validate recovered properties

    • Parse and validate the recovered values before updating runtime fields.
    • Invalid persisted configuration follows the existing recovery error-handling path rather than being silently accepted.
  5. Add regression coverage

    • Cover configuration persistence and recovery through Gson serialization/deserialization, legacy metadata compatibility, ALTER precedence, invalid configuration, repeated recovery, and Kafka/Kinesis job types.

Backward Compatibility

The fix uses the existing serialized jobProperties map and the original CREATE statement. It does not introduce new metadata fields or require a serialization-format migration.

For existing jobs whose metadata does not contain exec_mem_limit, the value can be recovered from the original CREATE statement when available.

This is particularly important during upgrades: an existing job may have been executing with an unintended default after an earlier FE restart. Once the fix is applied, its effective configuration can return to the value originally specified by the user.

The change is limited to configuration persistence and recovery. It does not intentionally modify job identity, Kafka partition offsets, ingestion progress, or task scheduling logic.

Release note

Fix Routine Load jobs unexpectedly reverting to default execution properties after FE restart or metadata recovery.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

Unit Test

Added RoutineLoadConfigRecoveryTest to cover:

  • Recovery of nondefault runtime properties after serialization/deserialization.
  • Recovery of Kafka and Kinesis Routine Load jobs.
  • Recovery of legacy metadata missing persisted configuration values.
  • Precedence of ALTER-modified properties over the original CREATE statement.
  • Handling of invalid persisted values.
  • Repeated serialization/deserialization.
  • Persistence of the resolved exec_mem_limit during job creation.

Existing Routine Load unit tests should also be executed to check for regressions.

Manual Test

  1. Start an unpatched FE and create a Routine Load job with:

    PROPERTIES (
        "max_filter_ratio" = "0.5",
        "exec_mem_limit" = "3333333333",
        "send_batch_parallelism" = "2",
        "load_to_single_tablet" = "true"
    )
  2. Pause the job and modify its filter ratio:

    ALTER ROUTINE LOAD FOR rl_recovery_legacy
    PROPERTIES (
        "max_filter_ratio" = "0.0"
    );
  3. Capture its configuration, job ID, state, and Kafka progress using SHOW ROUTINE LOAD.

  4. Gracefully stop the unpatched FE, preserve its metadata, and start the patched FE using the same metadata.

  5. Verify the recovered job retains:

    Property Expected value
    max_filter_ratio 0.0
    exec_mem_limit 3333333333
    send_batch_parallelism 2
    load_to_single_tablet true
  6. Verify that the job ID, paused state, and stored Kafka offsets remain unchanged.

  7. Create a new Routine Load job using the patched FE with the same nondefault properties. Restart the patched FE and verify that its configuration is also preserved.

Test results: Update the checkboxes and include actual execution results before submitting or marking validation complete.

  • Behavior changed:

    • No.
    • Yes.

    Routine Load jobs now recover their explicitly configured execution properties instead of silently reverting to defaults.

    For existing jobs previously affected by this issue, the effective execution settings may change on the first FE restart with the fix, restoring the user-configured values. This is the intended correctness improvement.

  • Does this need documentation?

    • No.
    • Yes.

    No new SQL syntax, configuration options, or user-facing APIs are introduced. This change restores the expected behavior of existing Routine Load properties.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Aranya7

Aranya7 commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28749 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6d2da1ee659410b268ca1899b0670eadba9b63a2, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17578	5126	5057	5057
q2	2113	300	244	244
q3	10280	1358	827	827
q4	4677	457	344	344
q5	7429	787	528	528
q6	181	185	150	150
q7	742	789	590	590
q8	9249	1234	1047	1047
q9	5435	4219	4202	4202
q10	6837	1282	1026	1026
q11	426	245	230	230
q12	634	394	282	282
q13	18090	3533	3006	3006
q14	275	263	248	248
q15	q16	704	723	678	678
q17	1192	749	584	584
q18	6888	6154	6093	6093
q19	1104	953	594	594
q20	385	334	222	222
q21	4703	2753	2535	2535
q22	367	328	262	262
Total cold run time: 99289 ms
Total hot run time: 28749 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5913	5891	5916	5891
q2	699	595	553	553
q3	5648	5744	5294	5294
q4	2232	2216	1543	1543
q5	5859	5786	5673	5673
q6	244	186	140	140
q7	2218	2006	1808	1808
q8	2816	2461	2481	2461
q9	7517	7472	7491	7472
q10	3635	3561	3141	3141
q11	581	402	387	387
q12	672	661	508	508
q13	3293	3623	3044	3044
q14	295	300	267	267
q15	q16	645	661	615	615
q17	7772	7013	6837	6837
q18	12969	12199	12842	12199
q19	921	799	784	784
q20	2124	2156	1907	1907
q21	5616	4568	4668	4568
q22	502	452	405	405
Total cold run time: 72171 ms
Total hot run time: 65497 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150970 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6d2da1ee659410b268ca1899b0670eadba9b63a2, data reload: false

query5	4313	597	461	461
query6	429	201	192	192
query7	4821	438	227	227
query8	326	164	152	152
query9	8771	3157	3116	3116
query10	454	311	260	260
query11	6025	3812	3556	3556
query12	147	93	87	87
query13	1274	466	330	330
query14	6167	3925	3654	3654
query14_1	3389	3368	3340	3340
query15	204	197	185	185
query16	943	409	383	383
query17	859	666	533	533
query18	2450	420	315	315
query19	200	167	138	138
query20	85	83	84	83
query21	213	130	109	109
query22	12916	13042	12866	12866
query23	13390	12898	12298	12298
query23_1	12619	12682	12584	12584
query24	7367	832	472	472
query24_1	479	472	491	472
query25	542	448	366	366
query26	1256	257	143	143
query27	2772	442	270	270
query28	4648	2003	2022	2003
query29	1579	601	462	462
query30	300	208	186	186
query31	1006	927	829	829
query32	148	93	91	91
query33	533	305	238	238
query34	951	858	507	507
query35	730	760	641	641
query36	821	807	750	750
query37	134	100	81	81
query38	1782	1762	1635	1635
query39	783	806	734	734
query39_1	722	722	734	722
query40	220	129	97	97
query41	73	68	68	68
query42	89	90	83	83
query43	342	360	314	314
query44	1342	707	743	707
query45	185	181	165	165
query46	857	972	602	602
query47	2853	2926	2809	2809
query48	312	314	226	226
query49	574	397	292	292
query50	716	279	208	208
query51	10791	10492	10603	10492
query52	78	86	67	67
query53	183	204	147	147
query54	235	192	174	174
query55	72	62	69	62
query56	245	226	212	212
query57	1671	1610	1583	1583
query58	279	258	248	248
query59	2183	2208	2020	2020
query60	301	246	232	232
query61	148	137	148	137
query62	388	338	280	280
query63	186	161	162	161
query64	2750	922	716	716
query65	3343	3273	3271	3271
query66	1797	433	303	303
query67	20312	19991	19987	19987
query68	2921	1015	608	608
query69	376	289	244	244
query70	838	759	779	759
query71	291	237	213	213
query72	2681	2627	2287	2287
query73	513	525	293	293
query74	4617	4486	4325	4325
query75	2336	2276	1901	1901
query76	2323	996	609	609
query77	345	389	271	271
query78	9815	9750	9151	9151
query79	1015	854	527	527
query80	752	439	334	334
query81	568	316	273	273
query82	583	133	106	106
query83	306	203	179	179
query84	314	123	97	97
query85	904	422	358	358
query86	347	240	229	229
query87	2009	1942	1808	1808
query88	3773	2757	2750	2750
query89	334	275	243	243
query90	1773	162	165	162
query91	158	150	125	125
query92	93	90	95	90
query93	1003	998	570	570
query94	517	301	266	266
query95	582	333	363	333
query96	679	478	235	235
query97	2455	2443	2324	2324
query98	161	152	159	152
query99	764	772	647	647
Total cold run time: 235037 ms
Total hot run time: 150970 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6d2da1ee659410b268ca1899b0670eadba9b63a2, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.10	0.10
query5	0.23	0.24	0.24
query6	1.17	0.67	0.67
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.41	0.34	0.36
query10	0.56	0.56	0.59
query11	0.19	0.14	0.14
query12	0.18	0.15	0.14
query13	0.49	0.50	0.49
query14	1.02	0.98	0.99
query15	0.61	0.60	0.60
query16	0.35	0.35	0.36
query17	1.12	1.12	1.09
query18	0.22	0.19	0.18
query19	2.04	2.06	2.01
query20	0.02	0.01	0.01
query21	15.48	0.18	0.13
query22	4.99	0.05	0.05
query23	16.29	0.32	0.13
query24	2.71	0.55	0.27
query25	0.11	0.06	0.05
query26	1.20	0.21	0.15
query27	0.05	0.06	0.03
query28	4.13	0.79	0.35
query29	12.48	4.19	3.23
query30	0.29	0.13	0.10
query31	2.77	0.53	0.32
query32	3.23	0.59	0.50
query33	3.17	3.17	3.16
query34	15.30	4.03	3.37
query35	3.33	3.33	3.34
query36	0.54	0.44	0.45
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.27 s
Total hot run time: 24.05 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 63.83% (30/47) 🎉
Increment coverage report
Complete coverage report

@morningman

Copy link
Copy Markdown
Contributor

/review

@morningman

Copy link
Copy Markdown
Contributor

Thank you, @Aranya7, for investigating this issue and contributing the fix and recovery tests.

I compared this PR with #70121, which was merged into master on November 5. That change addresses the recovery of all four properties covered here: max_filter_ratio, exec_mem_limit, send_batch_parallelism, and load_to_single_tablet.

The merged implementation restores the filter ratio, parallelism, and single-tablet setting from the persisted job properties, preserving an altered filter ratio. It restores exec_mem_limit from the original CREATE statement, which is sufficient because ALTER ROUTINE LOAD does not support changing that property.

I'm closing this PR as superseded by #70121. We appreciate the work you put into identifying the problem and covering the recovery scenarios. If you find a case that remains affected after #70121, please share it so we can follow up.

@morningman morningman closed this Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review recovery stopped: No final review submission was declared
Workflow run: https://github.com/apache/doris/actions/runs/42175734426

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Routine Load properties silently revert to defaults after FE restart

3 participants